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Abstract 

This paper discusses the results of a study comparing the psychometric qualities of 
two forms of an identical survey: one administered in paper-and-pencd format and the 
other adrninistered in Web format. The survey addressed the topic of college course 
anxiety and was used to survey a sample of vmdergraduate students (N = 36). The 
psychometric qualities investigated include the overall reliability (internal consistency), 
and item analyses of the 56 individual survey items for both the paper-and-pencil and 
Web surveys. With respect to reliability, results reveal a good deal of similarity between 
the two modes of survey delivery. Analysis of the paper-and-pencH survey data 
generated a Cronbach's a of .91; analysis of the Web survey data generated a value of 
.88. Further informal comparisons of the item analyses revealed very similar patterns in 
the responses to individual items, indicating that the two modes of survey delivery were 
quite comparable. 
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The Mouse or The Pencil? 



A Psychometric Comparison of Web-Based and 



Traditional Survey Methodologies 



Background 

The Internet has had a substantial impact on the field of survey research (Shannon, 
Johnson, Searcy, & Lott, 2001). This is evidenced by the growing niunber of 
electronically administered surveys over the past several years. Web surveys —which 
Dillman (1998) has referred to as one type of "self-administered" survey — are an 
extremely promising method of data collection (Schillewaert, Langerak, & Duhamel, 
1998). Young and Ross (2000) state that the use of the Internet to collect data may be one 
of the most profound developments in survey research. Witte, Amoroso, and Howard 
(2000) indicated that Internet research is "an area marked by great potential but also 
little experience" (p. 180). Specifically, the World Wide Web can be used as a resource 
for obtaining and efficiently processing large amounts of data, often the type collected as 
a result of administering a survey (Carbonaro & Bainbridge, 2000). Unlike more 
traditional survey delivery methods (e.g., mail and telephone), it is not clear what 
principles should guide the construction and implementation of electronic surveys 
(Shannon et al., 2001). 

Advantages of Web surveys include a high rate of response, short time frame for the 
collection of responses, and time and cost savings. The Web certainly addresses the need 
for a less expensive and more expedient method of data collection (Heflich & Rice, 1999; 
Schillewaert et al., 1998). Furthermore, several additional benefits of using the Web for 
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data collection have been identified (Carbonaro & Bainbridge, 2000; Mertler, 2001, 2002a, 
2002b; SchiUewaert et al., 1998; Shannon et al., 2001). These benefits include a faster 
response, protection against the loss of data, easy transfer of data into a database for 
analysis, cost savings, convenience for the respondent, the possibility of wider 
geographic coverage, and a potentially better response rate- although this "advantage" 
is not uniformly agreed upon by the community of survey researchers (Matz, 1999). 

However, it is important to note, as with any method of data collection, there also 
exist disadvantages. These include the potentially nonrandom nature of the sample, 
unavailability of population lists, computer access to the survey, and various 
technology-related issues. Additional limitations include the inability to clearly define 
the population, lack of technological familiarity on the part of respondents or their 
willingness to use a computer to complete the survey, the potential for being able to 
identify respondents, and browser incompatibility problems (Carbonaro & Bainbridge, 
2000; SchiUewaert et al., 1998; Shannon et al., 2001). 

One of the most substantial concerns about Web surveys is the potential nonrandom 
nature of the respondent group (Mertler, 2001, 2002a, 2002b; Witte et al., 2000). 

However, the issue of nonrandomness is not unique to Web-based survey research and 
can be addressed through the maintenance of an accurate list of population members, 
when feasible. Survey research professionals have suggested that Web surveys be used 
prunarUy with specifically identifiable samples such as "in-house" employee groups 
(Shannon et al., 2001). Alternatively, Taylor (2000) has suggested that we remember that 
onUne data collection is not based on probability sampling, but rather on "volunteer" or 
"convenience" sampling. ( 
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Beyond the strengths and weaknesses of this methodology, Carbonaro and 
Bainbridge (2000) have outlined several other issues with which researchers should be 
concerned. First, access to the survey must be as simple as possible for all respondents. 

The more complex the process of completing the survey, the lower the resultant 
response rate will undoubtedly be. Second, the process must be designed such that 
respondents of the Web survey are able to complete it with the same relative ease as if 
they had received a traditional paper version. Third, some sort of security system is 
required, in order to maintain the integrity of the data, but also to put at ease the mind 
of the respondent. Finally, completion of the survey must require only minimal 
computer skills — including the ability to use an Internet browser, enter a specific URL, 
use a mouse, and type on a word processor. Matz (1999) and Shannon et al. (2001) also 
point out that a Web survey must somehow be publicized. Some mechanism must be 
used to direct potential respondents to the actual URL containing the survey, either by 
providing a link to the URL in an email message or by providing the URL in a cover 
letter. 

There exists somewhat of a void in the research literature on the topic of electronic 
surveys, in particular, comparing their effectiveness to that of mail surveys 
(Underwood, Kim, & Matter, 2000). Often, the results of Web-based surveys differ when 
compared to written questionnaires and telephone surveys (Taylor, 2000), although 
Saphore (1999) foimd that there were no differences in the pattern of responses between 
a Web survey and an identical pencil-paper form of the same survey. Furthermore, he 
concluded that there were no differences in the psychometric qualities of the two forms 
of the survey. 
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In another comparison-type study, Matz (1999) compared the responses received 
from a Web survey with those received from a traditional paper-and-pencil survey. She 
found no significant differences between the demographic characteristics — such as age 
and gender — of the respondents completing the Web survey and those completing a 
mailed, written survey. Additionally, she found no significant differences between the 
content or pattern of responses for either group. The only significant finding -albeit, a 
substantial one — was that the overall response rate for the paper survey (43%) was 
significantly higher than that for the Web survey group (33%). A final result discussed 
by the researcher was that the paper instrument seemed to provide more flexibility to 
the respondents. They could freely make comments about items they did not 
imderstand or felt were ambiguous. In spite of these last two findings, she concluded 
that Web surveys seemed to be a reasonable alternative - or, at least, equally suspect— to 
mail surveys. 

Underwood et al. (2000) also conducted an empirical study comparing the 
characteristics of Web and mail surveys. When the data from the adrrdrdstration of the 
same survey via two different media were compared, the researchers concluded that 
women responded at greater rates than men, regardless of survey method, and that 
underrepresented minority students generally responded at lower rates than 
Caucasians, Asian Americans, and International students, also regardless of survey 
method employed. This led them to further conclude that the characteristics of 
respondents, rather than specific survey method used, are more closely related to 
response rates. 

Because of the technological skills required to develop and implement Web-based 
surveys, the driving force behind their use has been technology specialists (or at least 
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someone with a background in technology) and not survey professionals (Dillman & 

Bowker, 2001; Shannon et al., 2001). In order to harness the potential for using the 

Internet for the collection of valid and reliable data, those most knowledgeable about 

survey research methodology — specifically with respect to causes and consequences of 

survey error (Dillman & Bowker, 2001)— must take an active role in its development as a 

viable methodology for data collection. Specifically, research is needed in order to 

compare responses from Web surveys and those conducted in more traditional manners 

\ 

(Dillman & Bowker, 2001). 

Purpose of the Study 

The primary purpose of this study was to add to the knowledge base regarding the 
use of Web-based surveys as a viable means of collecting data for educational research. 
Specifically, the researchers investigated and compared the relative effectiveness, 
psychometric qualities, and response patterns of two versions of the same survey: one 
delivered as a Web-based survey and the other as a paper survey. A secondary purpose 
was to assess the specific methods used in this study for data collection in order to refine 
the procedures for future extensions of this line of inquiry. 

Methods 

In the Spring 2002 term, we contacted the directors of an introductory English course 
(writing composition) and an introductory Math course (statistics) to obtain permission 
to deliver surveys to randomly selected sections. We gained access to 15 writing sections 
and 18 statistics sections. From each course, we randomly selected 6 sections from which 
we recruited students. The reason for the selection of students in two courses was to 
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investigate (in a separate study) the existence of differential course anxiety. For purposes 
of this paper, only survey data resulting from the English class participants were used. 

Course instructors in the 12 selected sections delivered the surveys during class in 
one of two formats: either paper-and-pencil or via the Web. We provided instructors 
with a script from which to read after handing out letters to their students introducing 
the survey. Six of the twelve sections (three writing, three statistics) received the paper- 
based survey and the other six received instructions for accessing and completing the 
Web-based survey. Students were asked to complete the surveys within two weeks. 

Items from two instruments measuring student attitudes toward statistics, both 
called the STATS (Sutarso, 1992; Rogness, 1993) were coUapsed into one instrument 
which we called the Course Attitude Scale (see Figure 1). Duplicate items or items with 
wording similar to others were removed, for a total of 56 unique items in the final scale. 
AU items on the new scale were re-worded to be generic to any course, not just statistics. 
So, an item such as 'Tf I were not required to take a statistics course for my major, I 
wouldn't take one" was changed to read "If I were not required to take THIS COURSE 
for my major, I wouldn't take it." The purpose of the study reported here was to pilot 
test not only this new instrument, but also the dual modes of delivery. 



Insert Figure 1 about here 



For the two modes of survey delivery, analyses included the computation of internal 
consistency reliability (alpha) coefficients and individual item analyses. AU analyses 
were conducted using SPSS (v. 10). 
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Results 

Although the data resulting from the administration of the two surveys were 
analyzed for the entire group, the analyses were primarily based on formal techniques 
followed by informal comparisons of the results by the two subgroups — namely, the 
data resulting from the traditional paper-and-pencU method of delivery (n = 21) and 
those resulting from the Web form of the survey {n = 15). The return rate for the total 
survey was equal to 28%; the return rate for the paper-and-pencU administration was 
equal to 34%, while that for the Web administration was equal to 22%. It should be 
noted that the researchers recognize the fact that these are low return rates. This was 
likely due to the tuning and procedures used for data collection. 

Analysis of the overall scale comprised of 56 items for the entire group (N = 36) 
revealed an internal consistency measure (i.e., Cronbach's index of internal consistency; 
also known as the alpha coefficient, or a) equal to .90, indicating a high degree of 
reliability. The internal consistency measure for the "'paper" sample {a = .91) was qrdte 
similar to that for the "Web" sample {a = .88). 

The resiilting data were also analyzed at the level of the individual items. 
Specifically, the internal consistency of the overall scale with each individual item 
removed was determined. This is typically done in order to assess the contributions of 
individual items to the total scale. The resulting analysis reports an alpha (a) coefficient 
for the total scale minus the particular item. A "new" a coefficient (i.e., for the 55-item 
scale) that is lower than that for the 56-item scale indicates that the particular item did in 
fact contribute to the total scale; in other words, when the item was removed, the overall 
reliability decreased. In contrast, a "new" a coefficient that is higher than that for the 
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original scale indicates that the particular item did not contribute to the overall scale; in 
other words, when the item was removed, the overall reHabihty improved. 

Of the 56 items appearing on the survey administered in a traditional manner (i.e., 
paper-and-pencil), analyses of 13 items revealed improved internal consistency values 
when they were individually removed from the scale. The results of the individual item 
analyses for the survey adrninistered via the Web were similar. Of the 56 items, 19 
showed improved internal consistencies when removed from the scale. A summary of 
analyses of these individual items and the amounts of improvement in overall scale 
reliabilities are presented in Table 1 and Table 2, respectively. Furthermore, it is 
important to note that, upon informal comparison of the Hst of 13 items identified from 
the paper-and-pencil method and the 19 items identified from the Web delivery method, 
6 items were common to both lists. 



Insert Table 1 about here 



Insert Table 2 about here 



Upon further informal comparison between the two sets of results, it is also 
interesting to note that two items— namely Items 53 and 55— both demonstrated 
substantial amounts of change in the value for alpha (a) for their respective modes of 
survey delivery, providing further evidence of similar patterns of response. With respect 
to the paper-and-pencil data, these two items ranked second and first, respectively, in 
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terms of the greatest amounts of change in alpha; with respect to the Web data, they 
ranked first and third, respectively. 

Discussion 

Web-based surveys have begim to have a profound impact on survey research and 
are likely here to stay. Similar to previous studies — e.g., Underwood (2000), Matz (1999), 
and Saphore (1999) —this study has provided evidence of the similarities, at least with 
respect to psychometric qualities, between Web-based and paper-and-pencil survey 
methodologies. With respect to the primary purpose of this study, very similar values 
for internal consistency reliability were observed for the two versions of the survey. 

Although not identical, the patterns of responses — as identified by contributions to the 
overall scale — across the two modes of delivery were fairly comparable. With respect to 
the secondary purpose of this study, assessment of the procedures used for data 
collection revealed some problems. 

Although the various instructors agreed to participate, we were never sure of how 
many students actually received the information and survey since we based our return 
rates on class enrollments (e.g., if students were absent and did not receive the 
information, they were included as nonrespondents in the return rates). The students 
may not have been properly motivated to respond to their respective surveys. This 
could have been a substantial cause of the poor return rates for the two forms of the 
survey, as well as a factor in our inability to accurately calculate the return rates. 

Additionally, although we provided the instructors with scripts to be read to students, 
we could never be sure exactly how that part of the data collection was handled. 

Unfortunately, this was necessitated by the lateness in the semester when permission 
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from departments was obtained to approach instructors about participating in the study, 
and when individual instructors agreed to participate. 

Furthermore, it became apparent to us that soliciting participation for a Web-based 
survey should be done electronically, and not by providing the URL on a piece of paper. 

It is our belief that providing a URL on a piece of paper (e.g., in a cover letter) is not 
equivalent to handing participants an actual paper-and-pencil survey. The latter is a 
more "direct" form of survey delivery — it is easy for the participant to respond. When 
participants are directed to an electronic survey (e.g., on a Web page) in the mariner we 
used here, it takes some initiative on their part to go to a computer, type in the correct 
URL, and then complete the survey — it creates somewhat of an "extra" step when 
compared to the more traditional mode of delivery. Requesting the participation of 
individuals via email (for example) would allow researchers to include in the "cover 
message" a Link to the Web survey, thus making completion of the survey a much 
easier — and comparable — process. 

Research is beginning to demonstrate that a Web-based approach is a viable means 
of gathering survey data. However, further research must be conducted on this mode of 
delivery in order to pass judgment on its relative merits. There are several advantages to 
electronic surveys, in general. These include such things as cost and time savings, as 
well as the ease of transfer of responses into a database. Are these advantages over 
paper-and-pencil surveys great enough to "counteract" the problems associated with 
nonrandom samples? Are potential respondents comfortable enough with technology to 
respond to surveys online? Or do they remain apprehensive about potentially realistic 
concerns such as anonymity, confidentiality, and security of their provided information? 

If the answer to the latter question is "yes," do we as researchers end up with a 
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"differential" type of random sample resulting from respondent self-selection? This, of 
course, is a concern for any type of survey research, but can we be sure that we are not 
ending up with a sample whose characteristics are different from those realized through 
paper-and-pencU forms of surveys? To address these and related issues, further research 
is most certainly called for. 

Web-based surveys must adhere to the principles of sotmd survey design and 
implementation. In some instances, those principles may need to be adapted in order to 
"match" the methodology, but without jeopardizing the integrity of the data or risking 
the anonymity and confidentiality of the respondents. As this methodology is refined, it 
will undoubtedly be used more frequently to conduct scholarly research. Empirical 
studies, such as this one, are necessary in order to guide this process of methodological 
refinement. 
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Figure 1 

The Course Attitude Scale 
(Web version) 
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Welcome... 




and thank you for taking a few minutes to complete the... 



Course Attitude Scale 



DIRECTIONS: 

For each statement, please respond by indicating the extent to which each statement is true of you. Indicate your responses by clicking on 

appropriate button. 

When you have finished, click on the *SUBMIT* button at the bottom of the page 

to send your responses. 





ENGLISH 112 


MATH 115 


In which class did you hear about or receive this survey? 


I G 


20 



Thinking only about the class you checked above. 
please indicate the extent to which each of the following statements is true of you... 



NUMBER 


STATEMENT 


NOT TRUE 
OF ME 










VERY TRUE 
OF ME 


I. 


Learning in this course is easy for me. 


I o 


20 


3 0 


40 


50 


60 


2. 


I understand this course better than the majority of people in my class. 


I o 


20 


3 0 


40 


50 


60 


3. 


This course makes me anxious. 


I o 


2 0 


30 


4 0 


50 


60 


4. 


I like working with numbers. 


1 o 


20 


3 0 


4 0 


50 


60 


5. 


I enjoy working with calculators. 


1 o 


20 


3 0 


4 0 


50 


60 


6. 


I enjoy working with computers. 


I G 


20 


30 


40 


50 


60 


7. 


My mother likes/liked this subject, so I will. 


1 O 


20 


30 


4 0 


50 


60 


8. 


My father likes/liked this subject, so I wflL 


I Q 


20 


30 


40 


50 


60 


9. 


1 feel that this course is only for men. 


I O 


20 


3 0 


40 


50 


60 


10. 


1 feel that this course is only for women. 


1 G 


20 


3 0 


4 0 


50 


60 


11. 


This course is very useful for my major. 


I O 


20 


3 0 


40 


50 


60 


12 . 


This course will improve my research ability. 


I G 


20 


30 


40 


50 


60 


13. 


This course will be important for my career. 


1 G 


20 


30 


40 


50 


60 


14. 


I will be more competent in my subject area when I master this course. 


I O 


2 0 


30 


40 


50 


60 


15. 


1 can master this course with a great deal of effort. 


I G 


20 


30 


4 0 


50 


60 1 


16. 


I study for this course regularly even when there is no specific assignment 


1 O 1 


|20 


|30 


4 0 


50 


60 1 


17. 


I see my instructor when I do not understand something in this course. 


lO 1 


|20 


|3d 


40 


50 


60 


18. 


I ask questions in this course when I do not understand. 


lO 1 


|20 


|3 0 ||4 0 


50 


60 


19. 


I believe this course is a very interesting subject 


1 G 


20 


0 

0 


5 0 


60 


20. 


If this course were not required for my major, I would not take it 


I O 1 


[20 


3 0||4 0 


50 


60 


21. 


I do not like this course. 


I G 


20 


3 0j|4 0 


50 


• 60 


22 . 


I am able to understand this subject as well as any other subject. 


I G 


20 


3 0|40 


50 


60 
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23. 


1 1 try to do as little work as I can in this course. | 


1 lO ! 


20 


|30 


|40 ! 


i|5 0 


60 1 


24. 


I doubt that any teacher could make me more comfortable with this subject | 


1 1 o 


20 


30 


40 


50 


60 


25. 


The right instuctor could probably make me enjoy this subject. 


1 0 


20 


|3 0 


4 0 


js o 


60 j 




I feel insecure while taking exams. 


I 0 


20 


|30 


40 I 


js 0 


60 


27. 


I am confident that I could do work which required me to use computers. | 


1 0 


20 


|s 0 


40 I 


5 0 


60 


28. 


j No matter how hard I study, I will not do well in this course. | 


1 0 ■ 


20 


|3 0 


40 I 


|50 


60 




I think being enrolled in this course is undesirable. 


1 o 


20 


3 0 


4 0 


50 


60 


30. 


I would like to take more courses in this subject 


1 o 


20 


30 


40 


50 


6 0 j 


31. 


No matter who teaches this course, I feel I would never be good at it. 


1 o 


20 


3 0 


40 I 


50 


6 O 1 


32. 


I think this course is worthwhile for most professionals in my field. 


1 o 


20 


30 


40 


50 


60 


33. 


I have trouble thinking mathematically. 


1 o 


20 


30 


40 


50 


60 1 


34. 


I have trouble thinking creatively. 


1 o 


20 


3 O 


4 O 


5 O 


60 


35. 


I forget information that I actually know during an exam due to anxiety. 


1 o 


20 
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36. 


I have little self-confidence when it comes to using computers. 
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37. 


1 feel at ease doing the work for this course. 
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38. 1 


Given a choice, I woul d prefer not to have to work with this material. j 
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39. 


I think that I am good at this material. 
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I like working with words. 
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41. 


I become very nervous prior to taking an exam. 
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42. 


1 try to avoid circumstances where I have to use this material. 
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43. 1 


I would like to take another class in this subject even if I wasn’t required to do so. | 
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44. 1 


While taking a test, 1 feel calm. 
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45. 1 


No matter who teaches this course, I would never like it 
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1 think this course is an important part of my career education. | 
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47. 


The thought of woiking with a computer makes me nervous. 
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48. 1 


1 1 find it difficult to stop worrying about how I did on an exam when it is over. j 
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49. ]| 


1 1 think that using a computer is often more trouble than it’s worth. j 
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50. 


I wish that I didn’t have to take any course in this subject. 
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51. 


I like the thought of using computers. 
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52. j; 


I think studying this material is a beneficial experience. 
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53. 


The right instructor could probably make this course easy for me. 
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54. 1 


One of the parts of this course which I dread is having to use computers. 
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I get anxiety if I even think about working on this course material. |! 
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I would never consider taking this course as an elective. | 
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Submit Survey] 




Thank you for participating in this research study. 



If you have questions or comments about this survey, please feel free to contact us... 




Mark A. Earley, Ph.D. 
Bowling Green State University 
Bowling Green, Ohio 43403 

earlevmQ.b^net. hssu. edu 

Craig A. Mertler, Ph.D. 
Bowling Green State University 
Bowling Green, Ohio 43403 
mertler(a),h2net. hssu. edu 
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Table 1 

Identified Items and Recalculated Reliabilities for Item Analysis of Data Resulting from 
Paper-and-pencil Survey 



Survey Item Number 









Item 3 ^ 
Item 4 ^ 
Item 8 
Item 16 ^ 
Item 18 
Item 23 
Item 24 
Item 27 
Item 34 
Item 47 



Alpha Coefficient 
(If Item Deleted) ® 



.9053 

.9059 

.9063 

.9054 

.9056 

.9082 

.9114 

.9089 

.9057 

.9061 



Amount of Change 
In Reliability 



+.0001 

+.0007 

+.0011 

+.0002 

+.0004 

+.0030 

+.0062 

+.0037 

+.0005 

+.0009 










^ Alpha coefficient of the total scale was equal to .9052 (rounded to .91). 

^ These items were also identified following the item analysis of data resulting from the 
Web administered survey. 
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Table 2 

Identified Items and Recalculated Reliabilities for Item Analysis of Data Resulting from 
Web Survey 



Alpha Coefficient Amount of Change 

Survey Item Nvunber (If Item Deleted) ^ In Reliability 



Item 3 ^ 


.8813 


+.0002 


Item 4 ^ 


.8850 


+.0039 


Item 5 


.8858 


+.0047 


Item 6 


.8823 


+.0012 


Item 9 


.8814 


+.0003 


Item 10 


.8814 


+.0003 


Item 15 


.8842 


+.0031 


Item 16 ^ 


.8819 


+.0008 


Item 17 


.8847 


+.0036 


Item 22 


.8828 


+.0017 


Item 26 


.8879 


+.0068 


Item 31 


.8816 


+.0005 


Item 33 


.8902 


+.0091 


Item 35 


.8820 


+.0009 


Item 36 


.8845 


+.0034 


Item 48 


.8876 


+.0065 


Item 53 ^ 


.8927 


+.0116 


Item 54 ^ 


.8822 


+.0011 


Item 55 ^ 


.8901 


+.0090 



^ Alpha coefficient of the total scale was equal to .8811 (rounded to .88). 

These items were also identified following the item analysis of data resulting from the 
paper-and-pencil adrninistered survey. 
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